Embedded system, fast structured light based 3d camera system and method for obtaining 3d images using the same

ABSTRACT

Disclosed are an embedded system, a fast 3D camera system based on structured light, and a method for acquiring 3D images using the system. The embedded system includes a pattern generation module for generating the pattern in real time and a camera trigger module for converting a camera synchronization signal into a trigger signal and transmitting the signal to a camera. A transform module for transforming the pattern into a video signal and a communication module for receiving a command for projecting patterns and capturing images may be further included. The fast 3D camera system based on structured light includes an embedded system for generating a pattern in real time upon receiving a command for projecting patterns and capturing images, a projection device for projecting the pattern, and a camera for capturing the image of an object onto which the pattern is projected.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application a continuation of U.S. patent application Ser.No. 15/180,024, entitled “Embedded System, Fast Structured Light Based3D Camera System and Method for Obtaining 3D images Using the Same”,filed Jun. 11, 2016, now U.S. Pat. No. 10,200,679; which claims priorityunder 35 U.S.C. § 119 to Korean Patent Application No. 10-2015-0083122(Filed on Jun. 12, 2015), the entirety of all two of which are expresslyincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an embedded system for implementing afast 3D camera system based on structured light, the fast 3D camerasystem based on structured light, and a method for acquiring 3D imagesusing the system.

Description of the Related Art

Generally, technologies for reconstructing a 3D image of an object usinga camera may be largely classified into active and passive techniques.As a typical example of active techniques, there are laser triangulationand a structured light-based method. As a typical example of passivetechniques, there is stereo vision. Active techniques are mainly usedfor research or industrial purposes because they may achieve higherprecision than passive techniques.

Meanwhile, a 3D camera using structured light is a modification of astereo camera. Unlike a stereo camera, in which two identical camerasare used, a 3D camera based on structured light is configured such thatone camera is replaced by a projection device such as a beam projector.Accordingly, in a 3D camera system based on structured light, a patternis projected onto an object using a projection device, the object ontowhich the pattern is projected is captured using an image capturingdevice such as a camera, and 3D information is acquired by analyzing thecaptured image.

In other words, because a stereo camera system passively uses featuresextracted from an image but a structured light-based camera systemactively projects a pattern onto an object using a projection device anduses the projected pattern to distinguish features, it may have a fasterprocessing speed and higher spatial resolution. Due to these advantages,the structured light-based camera system is widely used in variousfields such as object modeling and recognition, 3D measurement,industrial inspection, reverse engineering, and the like.

However, conventional 3D sensors based on structured light have thefollowing problems.

First, if precise control on the nanosecond level is not achievedbetween a camera and a device for projecting patterns, a time delay mayoccur, but it is difficult to implement such nanosecond-level precisecontrol. Also, the longer it takes to project a pattern and capture animage, the longer it takes to perform 3D sensing, making it difficult toacquire 3D data of a moving object. Furthermore, if no embedded systemis provided, communication processing by the OS (Operating System) of aPC may incur an additional delay.

Alternatively, if an additional processor and an embedded device areused to control a camera and a projector, additional memory is requiredfor pattern codecs for projection. This may increase the size of theprocessor and embedded device and incur additional costs.

Besides, because a 2D sensor, especially for an industrial robot, has anintelligent solution but has low positioning accuracy, a lot ofcalibration processes are required. These calibration processes are notsuitable for modern automated systems, which rapidly change, and maypose an obstacle to the spread of industrial automation.

In other words, a 3D sensor used for an industrial robot must have atolerance of error less than 1 mm and fast processing time, but such asensor is not only very expensive but also large and heavy. For example,AICON's smartSCAN has high precision but the weight is 4 kg, the priceis 100,000 to 150,000 dollars, and the measurement distance ranges from6 to 26 cm. Due to the weight, it is difficult to mount on a hand of arobot unless the robot is an expensive robot with a high payload, andthus it is fixed to the surrounding environment. That is, because itremains in a stationary state, its usability suffers. As another exampleof a 3D sensor, there is Faro's Laser Scanner Focus3D, which has atolerance of 2 mm and weighs 5 kg. This sensor has low precision,although it is a laser sensor, and has the same problem as the AICON'ssensor due to the weight thereof. Meanwhile, Imetric's IScan M300 V70 isa lightweight mobile system that weighs 2.4 kg and has a measurementrange from 23.5 to 75 cm. This system is promoted as a lightweightmobile system for solving the above problems, but its weight is similarto or greater than the payload of a low-cost industrial robot, making itdifficult to mount on the hand of the robot.

In consideration of the above-mentioned problems, the implementation ofa high-precision, high-accuracy, and lightweight 3D sensor isnecessarily required in order to enable robots to replace humans incellular manufacturing and visual inspection, which have depended onhuman labor. Also, as techniques for accurately recognizing objectsusing 3D images have come to be widely used, the implementation of sucha 3D sensor is required.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a method and systemthat enable fast processing and cost reduction in a structured lightbased 3D camera. Specifically, the present invention intends to providea fast embedded system, a fast 3D camera system based on structuredlight, and a method for acquiring 3D images using the system at low costby providing precise control of image capturing using a camera, controlof the exact time at which a pattern is projected using a projector (ora device for projecting images through a digital micro-mirror device(DMD)), and real-time generation of a pattern to be projected and byadding a Field Programmable Gate Array/Digital Signal Processor(FPGA/DSP) or a Micro Controller Unit (MCU) to a system, rather thanadding additional memory to a PC.

The present invention provides a system and method in which a patternmay be generated in real time using an embedded system and a fast andlightweight 3D camera based on structured light may be implemented bysynchronizing a camera with a projection device.

According to an aspect of the present invention, a pattern may begenerated in real time in an embedded system rather than in a computingdevice, and the generated pattern may be transformed into a video signaland sent to a projection device. The embedded system includes a patterngeneration module for generating the pattern in real time and a cameratrigger module for converting a camera synchronization signal into atrigger signal and transmitting the trigger signal to a camera.

The pattern generation module may generate the pattern in real time inresponse to a command for projecting patterns and capturing images.

The command for projecting patterns and capturing images may be receivedfrom the computing device.

The embedded system may further include a communication module forreceiving the command for projecting patterns and capturing images.

The embedded system may further include a transform module fortransforming the generated pattern into the video signal in order toproject the generated pattern using the projection device.

According to another aspect of the present invention, there is provideda fast 3D camera system based on structured light, which includes anembedded system for generating a pattern in real time upon receiving acommand for projecting patterns and capturing images, a projectiondevice for projecting the pattern, and a camera for capturing an imageof an object onto which the pattern is projected.

The fast 3D camera system based on structured light may further includea computing device for sending the command for projecting patterns andcapturing images to the embedded system and reconstructing a 3D image byreceiving the captured image from the camera.

The computing device may include a user interface through which a userinputs a command and checks the captured image and the reconstructed 3Dimage and a communication module for sending the command for projectingpatterns and capturing images to the embedded system and receiving thecaptured image from the camera through a continuous communication link,which is either a wired link or a wireless link.

The computing device may further include a storage unit for storing thecaptured image and the reconstructed 3D image.

The camera may be configured to be synchronized with the projectiondevice through the trigger signal, to capture the image, which includesthe pattern projected by the projection device, and to send the capturedimage to a computing device via a standard communication link, which iseither a wired link or a wireless link.

According to a further aspect of the present invention, there isprovided a method for acquiring a 3D image using a 3D camera based onstructured light, in which a pattern is generated in real time in anembedded system rather than in a computing device, and in which thegenerated pattern is transformed into a video signal and is provided toa projection device. The method includes generating the pattern in realtime in the embedded system, and generating a camera synchronizationsignal in the embedded system, converting the camera synchronizationsignal into a trigger signal, and transmitting the trigger signal to thecamera.

The method may further include sending a command for projecting patternsand capturing images from the computing device to the embedded systemvia any one of a wired link and a wireless link.

The generating the pattern may be configured to generate the pattern inreal time in response to the command for projecting patterns andcapturing images.

The method may further include transforming the pattern into a videosignal in the embedded system in order to project the pattern onto anobject using the projection device.

The method may further include projecting the pattern onto the objectand capturing an image of the object onto which the pattern isprojected.

The method may further include reconstructing a 3D image from thecaptured image in the computing device.

The method may further include repeating the generating the pattern, thetransforming the pattern, and the transmitting the trigger signal anumber of times determined at an initial setup process.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and other advantages of thepresent invention will be more clearly understood from the followingdetailed description when taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a schematic block diagram of a 3D camera system based onstructured light;

FIG. 2 is a schematic diagram illustrating the internal configuration ofan embedded system;

FIG. 3 is a flowchart of the processes performed in a 3D camera systembased on structured light;

FIG. 4 is a schematic diagram illustrating an experimental system;

FIG. 5A to 5D illustrate an example of the process of generating apattern to be projected;

FIG. 6 shows a user interface screen in which a 3D image, reconstructedin an experiment, is displayed;

FIG. 7A shows the total time taken to capture an image when aconventional experimental method is used; and

FIG. 7B shows the total time taken to capture an image when anexperiment is performed according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention may be variously changed, and may have variousembodiments, and specific embodiments will be described in detail belowwith reference to the attached drawings. However, it should beunderstood that those embodiments are not intended to limit the presentinvention to specific disclosure forms and that they include allchanges, equivalents or modifications included in the spirit and scopeof the present invention.

In the present specification, a singular expression includes a pluralexpression unless a description to the contrary is specifically pointedout in context. In the present specification, it should be understoodthat tams such as “include” or “have” are merely intended to indicatethat features, numbers, steps, operations, components, parts, orcombinations thereof are present, and are not intended to exclude thepossibility that one or more other features, numbers, steps, operations,components, parts, or combinations thereof will be present or added.

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings. In the followingdescription of the present invention, the same reference numerals areused to designate the same or similar elements throughout the drawings,and repeated descriptions of the same components will be omitted.

FIG. 1 illustrates a schematic block diagram of a structured light-based3D camera system. Referring to FIG. 1, a structured light-based 3Dcamera system 100 includes an embedded system 103, a camera 104, and aprojection device 105. The system 100 may further include a computingdevice 101. Also, the system 100 may further include at least one of acommunication link 102 between the computing device and the embeddedsystem and a communication link 106 between the computing device and thecamera.

In the structured light-based 3D camera system 100, the computing device101 includes a user interface, which is capable of receiving input froma user in order to acquire a 3D image using the structured light-based3D camera and of checking image data, and a communication interface,which is capable of sending a command for projecting patterns andcapturing images to the embedded system and of receiving a capturedimage from the camera. Also, the computing device may further include astorage unit for storing initial settings, captured images,reconstructed 3D images, and the like. The computing device 101 may beany one of, for example, a PC, a laptop, a notepad, and various smartdevices. In the computing device 101, when a user inputs a command forprojecting patterns and capturing images through the user interface inorder to acquire a 3D image, the command may be sent to the embeddedsystem through the communication interface. Selectively, a user may setin advance the time at which a command for projecting patterns andcapturing images is to be sent and the number of times the command is tobe sent through the user interface. Also, the computing device 101 mayreceive a captured image, including the projected pattern, from thecamera 104, and reconstruct a 3D image from the captured image.

The communication link 102 is a continuous communication link and awired or wireless communication link for sending a command forprojecting patterns and capturing images from the computing device 101to the embedded system 103.

The embedded system 103 receives the command from the computing device101 and generates pattern data in real time without using the internalmemory thereof. Then, the embedded system 103 generates a pattern videosignal for projection and a camera trigger signal for capturing theimage of the object onto which the pattern is projected. The embeddedsystem 103, the camera 104, and the projection device 105 may bevariously arranged, and may be physically connected with each other.Through the physical connection, the pattern video signal and thetrigger signal may be delivered to the projection device 105 and thecamera 104, respectively.

The projection device 105 is a device that receives the video signal andprojects the pattern, such as a projector or the like.

The camera 104 includes a CCD-type or CMOS-type image sensor. The camera104 is synchronized through the trigger signal, whereby the shutterthereof is opened at the exact time and the open state is preciselymaintained for the predetermined time period in order to capture theimage of an object, including the pattern projected by the projectiondevice.

The communication link 106 between the computing device and the camerais a continuous communication link and a wired or wireless communicationlink for sending the captured image, which includes the projectedpattern, from the camera 104 to the computing device 101.

FIG. 2 illustrates a schematic diagram of the internal configuration ofan embedded system. The embedded system 103 may be implemented based ona Field Programmable Gate Array (FPGA), a Digital Signal Processor(DSP), a microcontroller, or the like. The embedded system 103 includesa pattern generation module 202 and a camera trigger module 205. Theembedded system 103 may further include a communication module 201 and atransform module 203. Also, the embedded system 103 may further includea software link 204, through which a camera synchronization signalgenerated in the transform module 203 is sent to the camera triggermodule 205.

The communication module 201 serves to receive a command for projectingpatterns and capturing images from the computing device 101, and may beimplemented through UDP/TCP, serial communication, and the like.

The pattern generation module 202 generates a pattern in real time whenreceiving the command for projecting patterns and capturing images fromthe computing device 101. This module may generate various patterns,which are calculated and processed using codecs in the embedded system103.

The transform module 203 transforms the generated pattern into the videosignal to be sent to the projection device. For example, the pattern maybe transformed into five VGA signals (Vertical synchronization,Horizontal synchronization, R Signal Data, G Signal Data, and B SignalData). At the same time, a camera synchronization signal, which is to besent to the camera trigger module 205 through the software link 204, isgenerated.

The camera trigger module 205 converts the received camerasynchronization signal into a trigger signal and transmits the triggersignal to the camera 104. The trigger signal is transmitted from theembedded system 103 to the trigger pin of the camera 104, whereby thecapture of an image by the camera can be perfectly synchronized with theprojection of a pattern.

FIG. 3 illustrates a flowchart of the processes performed in astructured light-based 3D camera system 100. Referring to FIG. 3, in thestructured light-based 3D camera system, the following processes areperformed in order to acquire the 3D image of an object.

A user may input camera-related values and parameter values through theuser interface screen provided by the computing device 101.Specifically, first, the initial value is set according to the kind ofthe camera 104 at step S301. Next, a camera parameter adapted to thestructured light-based 3D camera system 100 is input and adjusted atstep S302. Finally, a trigger is set at step S303 in order tosynchronize the capture of an image and the projection of a pattern.When the above input process has been completed, the camera waits for atrigger signal at step S304.

Then, the computing device 101 sets a communication port and acommunication speed, connects to the embedded system 103 at step S305,and transmits a predefined signal (that is, a command for projectingpatterns and capturing images) to the embedded system 103 at step S306.

The embedded system 103 receives the signal (the command for projectingpatterns and capturing images) and calculates pattern data at step S312.Then, the embedded system 103 transforms the generated pattern into avideo signal at step S313. For example, it may be transformed into fiveVGA signals (V Sync, H Sync, R Signal Data, G Signal Data, and B SignalData). Simultaneously with the transformation, a camera synchronizationsignal is generated. The generation of the pattern data, thetransformation of the pattern data and the generation of thesynchronization signal are repeated a predefined number of times at stepS315. The transformed pattern is sent to the projection device 105, andthe trigger signal (S317), generated from the synchronization signal, istransmitted to the camera 104 at step S314.

The projection device 105 receives the transformed pattern (S316) andprojects the pattern onto a target object at step S318, and the camera104 captures the image of the object, onto which the pattern isprojected, at step S319.

The camera 104 sends the captured image to the computing device 101 atstep S308 through the communication link 106 between the computingdevice and the camera. The computing device 101 stores the receivedimage in a storage medium at step S309 and constructs a 3D image bydecoding the stored image at step S310.

Meanwhile, a user may perform the following processes through a userinterface screen provided by the computing device 101. The user mayperform an initial setup process such as setting the initial value of acamera, inputting camera parameters, setting triggers, and the like.When the initial setup process is completed, the user may send a commandfor projecting patterns and capturing images to the embedded system byselecting a pattern projection/camera capture button or the like in theuser interface screen. Selectively, the user may set in advance the timeat which the command for projecting patterns and capturing images is tobe sent and the number of times the command is to be sent, whereby thecommand may be automatically sent the designated number of times at thedesignated time.

The user may monitor the processes performed in the embedded system 103,the camera 104, and the projection device 105 in real time through theuser interface screen.

Also, the user may check the object image, including the pattern, whichis received from the camera 104, through the user interface screen.Also, the user may check the 3D image of the object, acquired throughthe 3D image reconstruction process, in the user interface screen. Here,in order to increase the clarity of the reconstructed 3D object image,the brightness, chroma, and contrast of the image may be adjusted, andan unnecessary background may be deleted from the image.

Experimental Example

An exemplary system of the present invention, configured as describedabove, is implemented and compared with a conventional method. A laptopis used as the computing device, and the laptop and the embedded systemare connected with each other via a cable. The embedded system isimplemented as a Field Programmable Gate Array (FPGA), and a camera anda projector corresponding to the projection device are fixed in a mounttogether in order to conduct the experiment. Meanwhile, in order toimplement the conventional method, a connection between a PC and acamera and a connection between the PC and a projector are made withoutan embedded system. Also, a plaster figure, shown in FIG. 4, is used asa 3D object to be captured. The experimental example of the presentinvention generates a pattern in real time using Hierarchical OrthogonalCoding (HOC), by which the length of code signals is divided into a fewlayers and each layer includes orthogonal codes recursively (reference:S. Lee, J. Choi, D. Kim, et al., “Signal Separation Coding for RobustDepth Imaging Based on Structured Light”, Proceedings of the IEEEInternational Conference on Robotics and Automation, 2005, pp.4430-4436, and Korean Patent Application Publication No.10-2013-0000356). FIG. 5A to 5D show an example of the process ofgenerating a pattern to be projected. Referring to the example, first, agrid is generated as shown in FIG. 5A. In order to encode the grid intoa binary number, the grid is divided into sections as shown in FIG. 5B.Then, the grid is encoded into a binary number (black: 1, white(absence): 0) as shown in FIG. 5C. Finally, a pattern to be projected isgenerated based on the encoded value, as shown in FIG. 5D. When an imageis captured, the pattern generated through the above process isprojected, and the image of the object onto which the pattern isprojected is captured. The pattern projected in the experiment is anexample, and the system of the present invention may generate variouspatterns in real time in addition to the pattern shown in FIG. 5D. FIG.6 shows an image reconstructed from the captured image in the laptop.FIGS. 7A and 7B respectively show a screen in which the total time takento capture the image in the experimental example of the conventionalmethod is displayed and a screen in which the total time taken tocapture the image in the experimental example of the present inventionis displayed. In the experimental example, when the conventional methodis used, the time taken to capture the image is 1865 ms, but the time is1077 ms in the system of the present invention, that is, the time isreduced by about 57% compared to the conventional method.

Through the embodiments, it is confirmed that the present invention isadvantageous in that image processing and signal processing may bequickly performed by selecting a system in which a high-performanceMicro Controller Unit (MCU) or an FPGA/DSP is installed. In the case ofa conventional 3D sensor based on structured light, precisesynchronization may not be performed due to the limitation of the OS ofa PC. However, because the present invention does not install any OS onthe embedded system, signal processing may be quickly performed. Also,because the next pattern to be projected is generated while the currentpattern is projected, a fast speed may be maintained. Accordingly, alarge number of patterns may be projected in a short time. Consequently,fast processing enables the measurement of a quickly moving object in3D, and the measurement may be performed more precisely. Additionally,because patterns may be generated in real time in the embedded system,there is no need to use additional memory, thus reducing the weight andexpense of the system.

The embedded system, the fast 3D camera system based on structuredlight, and the method for acquiring 3D images using the system accordingto the present invention relate to an embedded system, the precision,the accuracy, and the speed of which are improved by applying the designand implementation of an embedded device based on a Micro ControllerUnit (MCU) or a Field Programmable Gate Array (FPGA), a fast 3D camerasystem based on structured light, and a method for acquiring 3D imagesusing the system. Accordingly, the present invention may solve problemsrelated to the weight and speed of the conventional 3D sensor based onstructured light.

According to the present invention, because a pattern is quicklygenerated in real time in the embedded system, it is not necessary toprocess the pattern in the OS of a computing device. Accordingly, thereis no need to use or allocate memory for the pattern in the OS of thecomputing device. Also, while a pattern is projected, the pattern to beprojected next is generated, thus maintaining a high speed. In otherwords, because the process of generating and sending a pattern isperformed in the embedded system, projection may be performed morequickly than when the pattern is generated by and sent from a PC.Therefore, the present invention may use more patterns than the existingmethod, thus achieving high precision. Alternatively, when the patternsthat are the same as those in the conventional method are projected, thespeed of the system may be improved with high real-time performance.

As described above, various embodiments of the present invention havebeen disclosed. Although specific terms have been used in the presentspecification, these are merely intended to describe the presentinvention, and are not intended to limit the meanings thereof or thescope of the present invention described in the accompanying claims.Therefore, those skilled in the art will appreciate that variousmodifications and other equivalent embodiments are possible from theembodiments. Additionally, because some of the steps described above maybe performed in any order, the steps may be performed in a differentorder. Also, some of the steps may be omitted, or other steps may befurther included.

What we claim:
 1. An apparatus for use with a computing device, acamera, and a projection device to reconstruct a 3D image of an object,the apparatus being separate from the computing device and comprising: apattern generation module to generate a first pattern; and a cameratrigger module to generate a trigger signal and transmit the triggersignal to the camera to synchronize a projection of the first patternonto the object by the projection device with an image capture operationof the camera.
 2. The apparatus of claim 1, wherein the patterngeneration module generates the first pattern in real time in responseto a command from the computing device.
 3. The apparatus of claim 1,wherein the pattern generation module generates a second pattern whilethe projection device projects the first pattern onto the object.
 4. Theapparatus of claim 2, further comprising a communication module forreceiving the command from the computing device.
 5. The apparatus ofclaim 1, further comprising a transform module for transforming thefirst pattern into the video signal for use by the projection device toproject the first pattern onto the object.
 6. The apparatus of claim 1,wherein the first pattern is not stored in a memory of the apparatus. 7.The apparatus of claim 3, wherein neither the first pattern nor thesecond pattern is stored in a memory of the apparatus.
 8. A method forgenerating a 3D image of an object the method comprising: generating, byan embedded system, a pattern in real time in response to a commandreceived from a computing device; generating, by the embedded system, acamera synchronization signal; converting, by the embedded system, thecamera synchronization signal into a trigger signal; and transmitting,by the embedded system, the trigger signal to a camera.
 9. The method ofclaim 8, further comprising: transmitting, by the computing system, thecommand to the embedded system via any one of a wired link and awireless link.
 10. The method of claim 8, wherein the pattern is notstored in a memory of the embedded system.
 11. The method of claim 10,further comprising: transforming, by the embedded system, the patterninto a video signal; and transmitting the video signal to a projectiondevice to cause the projection device to project the pattern onto theobject.
 12. The method of claim 11, further comprising: capturing, bythe camera, an image of the object having the pattern projected thereon.13. The method of claim 12, further comprising: generating, by thecomputing device, the 3D image of the object from the captured image.14. The method of claim 11, wherein the method is performed a pluralityof times according to an initial setup process.